Fast Parallel Training of Neural Language Models
نویسندگان
چکیده
Training neural language models (NLMs) is very time consuming and we need parallelization for system speedup. However, standard training methods have poor scalability across multiple devices (e.g., GPUs) due to the huge time cost required to transmit data for gradient sharing in the backpropagation process. In this paper we present a sampling-based approach to reducing data transmission for better scaling of NLMs. As a “bonus”, the resulting model also improves the training speed on a single device. Our approach yields significant speed improvements on a recurrent neural network-based language model. On four NVIDIA GTX1080 GPUs, it achieves a speedup of 2.1+ times over the standard asynchronous stochastic gradient descent baseline, yet with no increase in perplexity. This is even 4.2 times faster than the naive single GPU counterpart.
منابع مشابه
Development of an in-cylinder processes model of a CVVT gasoline engine using artificial neural network
Today, employing model based design approach in powertrain development is being paid more attention. Precise, meanwhile fast to run models are required for applying model based techniques in powertrain control design and engine calibration. In this paper, an in-cylinder process model of a CVVT gasoline engine is developed to be employed in extended mean valve control oriented model and also mod...
متن کاملIncremental Adaptation Strategies for Neural Network Language Models
It is today acknowledged that neural network language models outperform backoff language models in applications like speech recognition or statistical machine translation. However, training these models on large amounts of data can take several days. We present efficient techniques to adapt a neural network language model to new data. Instead of training a completely new model or relying on mix...
متن کاملA Hybrid Optimization Algorithm for Learning Deep Models
Deep learning is one of the subsets of machine learning that is widely used in Artificial Intelligence (AI) field such as natural language processing and machine vision. The learning algorithms require optimization in multiple aspects. Generally, model-based inferences need to solve an optimized problem. In deep learning, the most important problem that can be solved by optimization is neural n...
متن کاملAutoregressive Attention for Parallel Sequence Modeling
We introduce an autoregressive attention mechanism for parallelizable characterlevel sequence modeling. We use this method to augment a neural model consisting of blocks of causal convolutional layers connected by highway network skip connections. We denote the models with and without the proposed attention mechanism respectively as Highway Causal Convolution (Causal Conv) and Autoregressive-at...
متن کاملA fast and simple algorithm for training neural probabilistic language models
In spite of their superior performance, neural probabilistic language models (NPLMs) remain far less widely used than n-gram models due to their notoriously long training times, which are measured in weeks even for moderately-sized datasets. Training NPLMs is computationally expensive because they are explicitly normalized, which leads to having to consider all words in the vocabulary when comp...
متن کامل